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ABSTRACT 

Digital learning games are useful educational tools with high motivational potential. With the application of games for 
instruction there comes the need of acknowledging learning game experiences also in the context of educational 
assessment. Learning analytics provides new opportunities for supporting assessment in and of educational games. We 
give an overview of current learning analytics methods in this field and reflect on existing challenges. An approach of 
providing reusable software assets for interaction assessment and evaluation in games is presented. This is part of a 
broader initiative of making available advanced methodologies and tools for supporting applied game development. 
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1. INTRODUCTION 

Applied games are games serving a primary purpose that goes beyond the aspect of pure entertainment. Most 
common in this field are digital educational/learning games, which represent an increasingly used e-learning 
technology. Their highly motivating character makes them effective educational tools for creating authentic 
learning tasks and meaningful, situated learning (de Freitas, 2013). With the application of educational games 
for instruction and learning there comes the need of accounting for learning game experiences also in the 
context of educational assessment. Conventional educational measures are not suitable in the context of 
educational games, since they are usually highly invasive and compromise flow (Van Eck, 2006). 

Learning analytics (LA) is considered as a key approach providing new opportunities for learning 
performance measurement, assessment, and improvement in and of applied games. Translating LA to 
learning games combines two major trends in e-learning and stimulates new research questions in both fields. 
After introducing the general notion of LA (section 2), this paper provides an overview of current analytics 
research and developments in applied games (section 3). Based on a discussion of existing challenges, in 
section 4 an asset-based approach is outlined that supports realizing and integrating LA in game technologies. 
This approach shall foster a broader uptake of applied games and LA by the game industry and facilitate their 
broader adoption in educational practice. 


2. BASIC NOTIONS OF LEARNING ANALYTICS 

LA is defined by the Society of Learning Analytics Research (http://www.solaresearch.org/) as “the 
measurement, collection, analysis and reporting of data about learners and their contexts, for purposes of 
understanding and optimizing learning and the environments in which it occurs”. LA is used to empower 
learners, instructors, and educational institutions, as well as to support intelligent tutoring systems. LA 
consists in a multi-step, cyclical process of data collection and pre-processing, analytics and action, and 
post-processing (Chatti et al., 2012). Data collection and pre-processing involves the gathering of educational 
data from different learning tools or applications and preparing and translating it into an appropriate format. 
The analytics and action phase denotes the actual application of analytic methods (e.g. structure discovery, 
relationship mining etc. - for an overview see e.g. Baker & Siemens, 2014) to extract meaningful patterns 
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and information from the data and to make use of the obtained results, (e.g. visualization, feedback, 
recommendations, adaptation). Post-processing refers to the idea of continually improving analytics, by 
refining analytics methods, using new methods, including new data sources etc. Until now, a lot of work in 
LA has focused on researching the methods of data collection and analysis; only recently more intensive 
efforts on their application in educational practice are being made (Ferguson, 2012). 


3. LEARNING ANALYTICS IN APPLIED GAMES 

By recording user ( interactions on a micro level learning games produce large amounts of data that may be 
used for LA. In fact, all digital games use in-game mechanics to appropriately respond to gamers’ actions. 
These analyses, however, focus on assessing playing performance instead of learning (Baalsrud Hauge et al., 
2014). A crucial question is how to harness and make sense of game-based user data in an educationally 
relevant manner. The valorisation of analytics for learning game assessment has started only recently. By 
combining ideas from gaming analytics, web analytics and traditional LA it is possible to establish 
meaningful analytics for educational purposes. In general, two types of LA in applied games can be 
distinguished (Westera et al., 2014): In-game analytics (real-time) and post-game analytics (off-line). 

A post-game approach to LA refers to supporting summative learning outcome measurement or 
identifying general learning patterns, and - more rarely, though - to the evaluation of the game artefact. 
Serrano-Laguna et al. (2012, 2014) proposed a two-step generic approach for LA in educational games, 
which is applicable with any kind of different game. Generic traces are gathered from gameplay, including 
game traces (start, end. quit), phase changes (game chapters), input traces (mouse movements, clicks), and 
other meaningful variables (e.g. attempts or scores). These data give rise to reports with general and 
game-agnostic information, like the number of students who played a game, average playing time, game 
phases in which users stopped playing etc. Visually reported, this information provides initial useful 
information on how learners interacted with a game. In a second step, additional information may be 
extracted by letting teachers define assessment rules based on and combining generic game trace variables to 
obtain new information (e.g. setting maximum time thresholds). These rules are defined closely in line with 
each game to match the educational objectives. A more specific approach of realising post-game LA was 
presented by Westera et al. (2014), who have used correlational and regression analyses to investigate 
switching behaviour between game objects and activities as predictor for learning performance. 

In-game LA is designed into a game and usually serves two purposes: Providing teachers and learners 
analytics results as a basis for action (e.g. selection of educational resources, decision on additional support 
or learning tasks etc.), or realizing dynamic adaptation during game time. Such stealth or embedded 
assessment avoids disrupting game experience, since assessment is appropriately integrated in the game and 
carried out non-invasively (e.g. Bellotti et al., 2013; Snow et al., 2015). Stealth assessment is usually 
implemented by linking observable game behaviour with an underlying model of learning outcome, 
competence etc. and regularly updating the user model (e.g. based on Bayesian score models or 
Competence-based Knowledge Space Theory). In this way, learning can be monitored and fostered, for 
example by generating progress reports and selecting new game experiences (Shute et al., 2009). Continuous 
non-invasive assessment was implemented in the educational games developed in the ELEKTRA and 
80Days projects ( Kickmeier-Rust & Albert, 2010): In a nutshell, learner actions during a complex 
problem-solving situation are monitored and interpreted in run-time in terms of available and lacking skills, 
current competence and motivational state. The information coming from this assessment is used to trigger 
adaptive interventions from a menu of different intervention types and tailored to the individual’s current 
state and needs, in order to support and guide the learner in the game and learning task and to retain 
motivation (Kickmeier-Rust et al., 2011). Another example of in-game LA was presented by Baker et al. 
(2007), who realised skill assessment in an educational action game by using game events as evidence for 
users’ mathematical skills, and to analyse study gains in accuracy over time and in speed over time with 
learning curves. Such kind of approach proved useful for formative assessment in educational games and 
may also be used to inform re-design and improvement of intelligent tutoring systems. Ventura et al. (2014) 
outlined a method for assessing persistence in educational games based on the time that learners spent on 
unsolved problems; this information may be used to tune gameplay difficulty, feedback and hints. 
Stephenson et al. (2014) elaborated an automated detector of engaged behaviour in a simulation game. The 
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aim thereby was to identify and model which learner actions give evidence of user engagement and. in the 
end, are predictive for success in the game. An integration of an engagement detector in the game may enable 
reporting the results back to learners and teachers for reflection. 

To summarise , LA has started to spread into the field of learning games, which is in fact a development 
that strengthens the position of both types of technologies. Nevertheless, there are still several challenges 
ahead, some of which are discussed in the next section. 


4. EXISTING CHALLENGES AND AN ASSET-BASED SOLUTION 
APPROACH 

Despite the theoretical and empirical evidence for the potential of educational games, there is still some 
reluctance among teachers towards their broader take-up and use in educational practice; for example 
because assessment routines built-in in educational games are usually black boxes and not tangible for 
teachers (Serrano-Laguna et al., 2012). Likewise, leisure game developers are hesitant to enter the applied 
games market, due to the high effort involved in learning game development and the difficulties of measuring 
learning outcomes. There is a need to make available approaches for transparent and reliable assessment in 
educational games - approaches that are based on valid assessment models, that are easy to use and provide 
meaningful educational information, and that give game industry evidence on the quality of a game. To 
maximise the potential benefits of LA for learning with applied games, LA actually should be incorporated 
already in the design phase of an educational game (Baalsrud Hauge et al., 2014). A great challenge with LA 
in educational games is the wide variety of different games available, which complicates the development of 
generic analytics tools that are applicable independent of a concrete game. The technical challenges of 
providing technologies for data collection, aggregation, analysis, and visualisation are therefore high. 

The European project RAGE (http://rageproject.eu/) aims at making available interoperable 
methodologies and tools for supporting applied game development. An applied gaming ecosystem will 
provide centralised access to reusable software components (so-called ‘assets’). Among others, assets for 
user data analytics are developed that can be used to gather and analyse interaction data from games and to 
integrate automated assessment and adaptation mechanisms in them. Assets for on-line and non-invasive 
interaction assessment are implemented that enable game developers to easily specify domain models, data 
collection, and interpretation rules as a basis for competence -based and motivational assessment. These 
assessment routines will empower learning game environments by providing meaningful input for game 
balancing assets supporting adaptation to individuals’ competence and motivational states. 

Existing analytics approaches in educational games often focus on realising continuous assessment for 
automatic live adaptation. The application of LA should be strengthened with respect to feeding back the 
information on skills acquired and learning progress to learners and teachers in a suitable way. In RAGE LA 
results also feed into dashboards and visualisation assets for users. These may further leverage the 
educational value of LA and may potentially be transferred to educational actions outside a gaming context. 

In addition to supporting LA for measuring and reporting learning success and for dynamic adaptation, 
the asset-based approach of RAGE shall also serve assessing and improving an educational game itself. An 
asset is elaborated to enable in-game evaluation. This evaluation asset thus represents an instrument for 
continuous evaluation of the quality of learning games by providing insights to users’ perception of games 
and their progress towards game goals. This is done by translating log data into meaningful information 
about game quality, user experience, and learning based on pre-defined, configurable evaluation metrics. The 
asset will facilitate the use of analytics for game evaluation purposes and will advance evaluation methods of 
applied games towards a meaningful triangulation of different data sources. The RAGE technologies will be 
exemplified, tested, and evaluated in asset-based games (mobile and desktop implementations) targeting 
employability skills in the context of different application scenarios. 

There is still more work to do to fully exploit the potential that LA in educational games may bring to 
optimize learning experiences. Beyond addressing the need for advanced, controllable, interoperable, and 
flexible technologies that facilitate the integration of LA in games, it needs to be taken into account, in 
particular, that games may be part of multiple learning activities and tools that learners carry out and use in 
parallel and, potentially, on the same educational objective or domain (Miller et al., 2014). 
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